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We study a two species fermion mixture with different populations on a square lattice modeled by a Hubbard 
Hamiltonian with on-site inter-species repulsive interaction. Such a model can be realized in a cold atom system 
with fermionic atoms in two different hyperfine states loaded on an optical lattice and with tunable inter-species 
interaction strength via external fields. For a two-dimensional square lattice, when at least one of the fermion 
species is close to half-filling, the system is highly affected by lattice effects. With the majority species near 
half-filling and varying densities for the minority species, we find that several correlated phases emerge as the 
ground state, including a spin density wave state, a charge density wave state with stripe structure, and various 
p-wave BCS pairing states for both species. We study this system using a functional renormalization group 
method, determine its phase diagram at weak coupling, discuss the origin and characteristics of each phase, and 
provide estimates for the critical temperatures. 
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Introduction - Experiments with ultra-cold atoms have re- 
alized mixtures of two different species of fermionic atoms 
with different densities, including mixtures of cold lithium 
atoms ( 6 Li) with different populations for two different hy- 
perfine states|Q]-|6). Fermionic systems with imbalanced spin 
populations have been studied in electronic materials, such as 
magnetic-field-induced organic superconductorsG] [8). Mix- 
tures of different species of fermions with unequal pop- 
ulations have also been considered in the study of quark 
matter[9 |. With the rapid experimental advances in the field 
of cold atom physics, these systems have the advantage of a 
great degree of tunability and control of inter-particle inter- 
actions, dimensionality, confinement, as well as number of 
pseudo-spin species. Imbalanced mixtures of cold fermionic 
atoms have attracted great interest due to their possible 
rich phase diagram^. Several phases have been observed 
experimentally|5] [6), such as imbalanced superfluid phase, 
phase separation, and normal Fermi liquid behavior. Also, 
Larkin-Ovchinnikov-Fulde-Ferrel (LOFF) state[10|, which 
involves Cooper pairs with finite center-of-mass momentum, 
and breached pair lfTTl fL2l state with zero center-of-mass mo- 
mentum, could be possible phases in special regions of the 
phase diagram lfl"3ll . In addition, there are studies of p-wave 
triplet pairing, caused by effective attractive intra-species in- 
teraction, proposed for strongly imbalanced cases in two lfl4l 
and in three lfT5l[l6ll dimensions. In addition to being cooled 
and trapped, fermionic atoms and mixtures can now also be 
loaded onto optical lattices lfT71[T8l where the interaction and 
the hopping strengths can be tuned, and effects of their inter- 
play with lattice geometry and dimensionality can be probed. 

In this Letter, we consider a two-component mixture of 
fermionic atoms with imbalanced populations on a two- 
dimensional square lattice. It is well known that the presence 
of a lattice can provide interesting strong correlation effects, 
such as spin density wave (SDW) and charge density wave 
(CDW) phases for fermions on a square lattice at half-filling. 
Extensive studies can be found in the literature for the case 
of balanced, SU(2) symmetric fermions, such as the Hubbard 
model[ll9! in various lattice geometries and fillings ll2"Tl|2"5l . 



Due to nesting of the Fermi surface (FS), SDW is the dom- 
inant instability for the repulsive Hubbard model on a two- 
dimensional square lattice at half-filling, followed by d x 2_ y 2- 
wave superconductivity when the system is doped away from 
half-filling i23l l24l . When nesting is completely destroyed 
by doping, the system becomes a normal Fermi liquid, aside 
from Kohn-Luttinger instabilities ll26l at extremely low tem- 
peratures. However, when the spin populations are unequal 
and SU(2) symmetry is broken, both the SDW and the singlet 
pairing will eventually be precluded due to the mismatch of 
the FS for the up and down spin fermions at strong population 
imbalance. From the SU(2) symmetry argument above, sin- 
glet pairing will be allowed and dominant in attractive cases 
when the filling of both species are still close to each other. As 
the Fermi surfaces become increasingly mismatched, the sys- 
tem is expected to be dominated by other instabilities or be- 
comes a two species Fermi liquid. If the inter-species interac- 
tion is initially repulsive, one expects to find SDW phase near 
the balanced case, switching to other and potentially richer 
correlated behavior as the polarization increases. Here we 
study this behavior using a weak-coupling functional renor- 
malization group (fRG) method[22, 24 1, and obtain the phase 
diagram for this system, focusing on the case where the ma- 
jority species stays close to half-filling and the density of the 
minority species is varied. The phase diagram contains sev- 
eral new phases, including a CDW phase and triplet pairing 
phases for both species. From our fRG study, we also obtain 
estimates for the critical energy scales for the different insta- 
bilities. 

For a microscopic interaction which is on-site, there is no 
bare intra-species interaction due to Pauli exclusion principle, 
but an effective long-range interaction can be induced via scat- 
tering between species. Previous studies[14- 16] have consid- 
ered such mediated interactions for mixtures of fermion gases 
(no lattice) with imbalanced populations, finding an attractive 
effective intra-species pairing interaction, leading to p-wave 
pairing of the majority species. In this study, we consider the 
effects of the interplay of interaction, population imbalance, 
and lattice effects. We show that lattice effects in particular, 
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FIG. 1: (a) RG flows of leading eigenvalues of different channels when system is balanced at half-filling, m = n± = 0, and the corresponding 
eigenvectors (M=28). (b) RG flows for density stripe phase at m = 0, /ij. = 0.01i. The snapshot of (c) intraspecies vertices Uff(m — 1, ri2, w.3) 
and (d) interspecies Ufi (ni = 1 , 712 , n% ) of fj,f — fii = near its critical scale, (e) Intraspecies and (f) interspecies are the same plot for fif — 0, 
fj,_l — O.Oli. The patch indices m — ~ 27 are show in Fig. |2|i). 



not only lead to a much richer phase diagram than that of a 
imbalanced mixture of fermionic gases, but also with a much 
higher transition temperature, even at weak interaction cou- 
plings, therefore more easily accessible to experimental ob- 
servation. 

1. The model and the fRG method. We consider the one- 
band Hubbard model for each species {a =f , \) of fermion 
(with creation operator ct. ) on a two-dimensional square lat- 
tice, and with on-site s-wave inter-species interaction Uq. The 
Hamiltonian can be written as: 

^ = £<rk C ka C ka + T/ X/ C k+qt C k'-qi C k'; C kt W 
crk kk'q 

, where £ CT k = — 2t CT (cosfc x + cosk y ) + \i a and 1/ is the 
volume of the system (hereafter set to be equal to 1). The dif- 
ferent chemical potentials \i a determine the densities and the 
hopping amplitude t a can be tuned by the optical lattices. In 
this Letter, we only consider the case tj- = t\, = t, and weak 
repulsive inter-species interaction (Uq > 0). There is previ- 
ous studv ll20l that shows results of attractive interaction. We 
neglect the confinement potential for the first approximation. 

The fRG approach has been applied to the study of the sta- 
bility and instabilities of Fermi liquids ll22l and has been ap- 
plied to a number of different lattice models [ 23-25 1. Start- 
ing with a microscopic model, the RG method provides an 
effective theory for low energy scales by integrating out 
high energy degrees of freedom, reducing the UV cut-off 
A. We perform a zero-temperature one-loop renormalization 
calculation for the interaction vertices f7 crCT /(ki, k2, ka, ki), 
where ki and k2(k3 and k-i) are the momenta of the incom- 
ing(outgoing) fermions, and ki + k2 = kj + k4, shown in 
Figj2ja). Figs|2|b)-(f) show the Feynman diagrams that con- 
tribute to the renormalized four-leg vertices at one-loop. In or- 
der to solve the RG equations, we discretize both FS (er =f , I) 
into M patches and integrate the renormalization group equa- 



tions numerically. We set the cut-off to be A; = 6te l , 
dl = 0.1 for an RG step, U = 2M and M = 28 in all cal- 
culations shown here. The bare interaction has no dynamics 
initially and we neglect self-energy corrections as justified for 
the one-loop limit ll22l l24l . After obtaining RG flows of all 
marginal interaction vertices that involve states near the Fermi 
surfaces, we extract vertex combinations that define specific 
instability channels at each RG step. The pairing channels are 
VBC?S ta) (k, q) = U n{n) (k, -k, q, -q), and: 

V B cs 3 , t0 (k, q) = j(Z7 t4 .(k,-k,q,-q) ± E^ t (k, -k,q, -q) 
+U n {-k, k, q, -q) ± C7 4t (-k, k, q, -q)) 

where the upper(lower) sign stands for singlet(triplet) pairing. 
Density wave channels from nesting interactions are given by: 

VsDW z ,CDw{K<l) = 

i(C/ n (k,q,k + Q,q+Q)+U u (k,q,k + Q,q + Q) 
TU n (k, q, k + Q, q + Q) T U |t (k, q, k + Q, q + Q)), 

where the upper(lower) sign stands for SDW Z (CDW) channel, 
and Q = (±7r, ±7t) is a nesting vector. The pseudo-spin in- 
dex a refers to two different hyperfine states of the atoms, thus 
the term SDW is understood as ordering of the pseudo-spin, 
and not of real spin. Also, the cold atoms are neutral particles 
and CDW refers to the number density (not electric charge), 
but we keep the CDW terminology in analogy to electronic 
systems. At each RG step, we diagonalize these M x M ma- 
trices as £K(k,q)0i n) (q) = \ { a n) cj> ( a n) (k) . As the leading 
eigenvalue of a defined channel diverge, we identify it as an 
instability of the system (the BCS pairing instability requires 
a negative eigenvalue). The form of the corresponding eigen- 
vector determines the symmetry of the order parameter of the 
instability. 
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FIG. 2: (a) Definition of interaction vertex, (b)-(f) The one-loop 
RG corrections to interaction vertices, (g) The RG flow (maker 
skipped) for majority pairing BCSf(solid line) and minority pair- 
ing BCS^dash line) at differnt chemical potential when majority is 
exactly half-filled, p-j- = 0. The FS for different filling are shown in 
(h) hi = 0.8t and (i) ^4. = 1.5t, also the patch indices n — 0~27. 



2. Nearly balanced case. In order to maximize lattice ef- 
fects, we focus on the case where the majority species is at 
or close to half-filling, ^ ~ 0, and we vary When the 
system is balanced (jj^ = jtij. = 0), SDW is the dominant insta- 
bility (RG flows shown in FigJTJa)). With increasing fi± the 
SDW Z order persists for the nearly balanced region (Fig(3]l. 
Fig ITJc) and (d) show intensity plots for Uf^(k%, k%, k^) and 
[/♦i {ki, k 2 ,k 3 ), at a given RG step, as k 2 and fc 3 are varied, 
covering all patches around the FS (patch index n from to 
27), with ki fixed at patch m = 1. There is an enhanced ver- 
tical line at 713 = 19 in these two figures since these patches 
have k% = k 3 + (it, it), that is, they are related by a nesting 
vector, and enhanced diagonal lines corresponding to nesting 
between k 2 and k 3 in Fig. 1(d). 

3. Weak imbalance. As the minority density is further de- 
creased away from half-filling, some inter-species vertices in- 
volving nesting vectors, such as umklapp processes, are no 
longer near the FS and are thus suppressed. However, there 
are still some non-perfect nesting particle-hole processes with 
net momentum equal to a reciprocal lattice vector, (2ir, 0) or 
(0, 2tt), which are allowed even under imbalance. An exam- 
ple of such a process is for a minority fermion to scatter from 
k^k + Qi — 8 across two opposite sides of its FS (say 
along the 45° direction), while a majority fermion scatters 
from q — » q + Q2 + S across the other two opposite sides 



of its FS (say along the 135° direction). With Qi = (it, tt), 
Q2 = ( _7r i 7r), and S = (<5, S) accounting for the FS mis- 
match, the net momentum for this process is (0, 2ir) and there- 
fore of umklapp-type and allowed by momentum conservation 
in spite of the imbalance. These vertices can still renormalize 
significantly under RG flow, and more importantly, they me- 
diate intraspecies nesting processes for the majority fermions, 
which do have a perfectly nested FS, so these mediated pro- 
cesses have the strongest flow. From a real space picture, 
with increasing imbalance, both species also have more empty 
sites to move around to minimize energy and, since the medi- 
ated interactions are attractive, this leads to a crossover in the 
RG flows from the SDW Z to a CDW phase. Figfjjb) shows 
the RG flow close to this crossover, where SDW 2 and CDW 
channels are almost degenerate. The lower panel of FigJTJb) 
shows the corresponding order parameter symmetries. The 
CDW channel of both species is doubly degenerate. For one 
of them, shown in FigfTJb), the order parameters for "J" and 
I are non-zero on two-opposite sides of the FS (second and 
fourth quadrants), and zero on the other sides (first and third 
quadrants), with the situation reversed for the other degener- 
ate channel. These correspond to stripe charge order in the 
diagonal direction. Figs[TJe) and[TJf) show snapshots of the 
interspecies and majority intraspecies vertices as the insta- 
bility is approached. Comparing with the SU(2) symmet- 
ric case (Fig[TJc) and |TJd)), where the interspecies nesting 
interaction is the dominant one, when the imbalance is in- 
creased (/xj, = O.Oli), the inter and intraspecies nesting ver- 
tices are comparable and carry different signs, indicating that 
this CDW stripe phase originates from weaker spin fluctua- 
tions caused by imbalance in addition to screened intrapecies 
attraction. Since it involves both spin and charge fluctuations, 
the size of this region and its critical energy scale depend on 
the filling fraction of both species (see Fig[3]). 

4. Strong imbalance. As the imbalance becomes stronger, 
the FS mismatch precludes all nesting processes as well 
as zero-momentum singlet pairing. Since the induced in- 
traspecies interactions are always attractive, the dominant RG 
flows present another crossover, from CDW stripe phase to 
triplet p-wave BCS pairing. The triplet pairing is for a single 
fermion species but it is generated by the initial bare inter- 
species interaction, the lattice (FS shape), and the imbalance 
(FS mismatch). Whether the dominant pairing is for the ma- 
jority or for the minority species depends on the interplay of 
these factors. Fig. |2]shows RG flows for three different minor- 
ity chemical potentials. As [i± increases, the minority pairing 
channel is dominant at first. This is because the nearly half- 
filled majority FS provides a large phase space in RG process 
to mediate and renormalize the minority intraspecies BCS ver- 
tex. The Feynman diagram in Fig. |2|c), which contains an 
internal fermion loop, gives a first nonzero correction to the 
intrapecies vertex from two interspecies vertices, of the form, 



d l u2,(k u k 2 ,k 3 ) = 



l-l(l + X)S a 



P P h{Ui,U 2 }, (2) 



where X denotes the operation XF(1, 2, 3, 4) = F(2, 1, 3, 4) 
and /3 ph {U u U 2 } = Tl{Ut, U 2 } + Tn{U u U 2 }, with T the 
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time-reversal operator and 

n{^i,^ 2 }=2 B p^i^. < c W k i» < l» k 3)^ k 4» < l' k a) (3) 



where B p h is an integral over an angular sector 12411 . From Eq. 

the induced intraspecies pairing interactions come from 
integrating out two interspecies interactions. Because the FS 
of the majority species is fiat, there will be more renormaliza- 
tion corrections to intraspecies interaction between minority 
species at first. For example, in Fig. |2ji), the momentum q at 
fixed magnitude can be translated anywhere along that branch 
of the FS. Fig. [3] shows that the A c for minority pairing de- 
creases monotonically with increasing polarization, because 
the phase space of induced minority intraspecies pairing be- 
comes smaller. This makes the majority pairing become the 
dominant instability. The induced pairing interaction of ma- 
jority species couples to its own nesting channel (k and k', 
for example, are connected by a nesting vector in Fig. |2jh)). 
Although the initial mediated majority intraspecies pairing in- 
teraction is smaller than that for minority species, as discussed 
above, the strong RG flow through a nesting channel leads to 
a majority pairing instability. We note that this majority su- 
perfiuid is not the same as proposed in previous studies of 
fermion gases ifT4l4T6ll . but is instead a lattice effect and has a 
much larger energy scale and critical temperature. As the den- 
sity of minority species is further decreased, it starts to behave 
like a fermion gas, with a small, almost circular FS, as shown 
for example in Fig. |2|i). When the density of minority species 
becomes less than quarter filled, majority pairing vertices in- 
volving fermions on opposite sides of the nested FS can no 
longer be mediated by minority fermions near the minority 
FS since this momentum would not be conserved. Since the 
majority species is still perfectly nested, as discussed before, 
there is more phase space for the minority pairing interactions 
to be mediated and renormalized. For large minority chemical 
potential, the critical temperature is much smaller, and even- 
tually reaches our numerical limit (A; ~ 10~ 5 *). Before this 
limit is reached, the flow of minority pairing is stronger than 
majority pairing and other channels. Although we do not ob- 
tain a divergence in the minority pairing channel, the system 
is expected to have a Kohn-Luttinger instability of minority 
pairing in a rather low critical temperature based on the phase 
space argument above. 

5. Conclusion. We have performed a fRG study of a pop- 
ulation imbalanced fermion system on a square optical lat- 
tice in two dimensions. When the majority species is close 
to half-filing, both density wave and triplet pairing phases are 
highly favorable due to the nesting of majority species FS. 
At the weak imbalance region, the competition between spin 
fluctuations and intraspecies attractions leads to a stripe den- 
sity wave phase, rather than the usual SDW phase. As the 
imbalance is further increased, p-wave superfiuid phases be- 
come dominant due to the mismatch of the FS. The competi- 
tion between majority and minority pairing is determined by 
their FS geometry and the mediated intraspecies pairing in- 
teraction coming from the initial bare interspecies on-site in- 
teraction. Both the stripe density wave phase and triplet su- 
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FIG. 3: The estimated critical temperature, A c = 6te of each 
dominate phase of repulsive model at different majority fillings. 



perfluid phases are enhanced by nesting of the majority FS, 
leading to much higher critical temperatures than that found 
for the imbalanced Fermi gas case. According to our calcula- 
tion, the critical temperature of stripe density wave is roughly 
k° DW (m = 0,/ij. = 0.01*) « 0.18* and that of p-wave su- 
perfiuid is Af CS ^ Of = 0.0003*, m = 0.03*) 0.016*, where 
hopping amplitude * can be controlled experimentally. 

Acknowledgments - We thank Pochung Chen for the com- 
putational resources in National Tsing Hua University, Tai- 
wan. SWT thanks Tun Wang for useful discussions in the 
early stages of this work. We gratefully acknowledge support 
from NSF under Grant DMR-0847801 and from the UC-Lab 
FRP under Award number 09-LR-05-1 18602. 



* Electronic address: shanwent@ucr.edu 
[1] M. W. Zwierlein et al, Science 311, 492 (2006). 
[2] G. Partridge et al, Science 311, 503 (2006). 
[3] M. W. Zwierlein et al, Nature (London) 442, 54 (2006). 
[4] Y. Shin et al, Phys. Rev. Lett. 97, 030401 (2006). 
[5] G. Partridge et al, Phys. Rev. Lett. 97, 190407 (2006). 
[6] Y. Shin et al, Nature 451, 689 (2008). 

[7] S. Uji et al, Nature 410, 908 (2001); ibidem., Phys. Rev. Lett. 

97, 157001 (2006). 
[8] W. A. Coniglio et al, Phys. Rev. B 83, 224507 (201 1). 
[9] R. Casalbuoni and G. Nardulli, Rev. Mod. Phys. 76, 263 (2004). 
[10] P. Fulde and R. Ferrel, Phys. Rev. 135 A550(1964); A. I. Larkin 

and Yu. N. Ovchinnikov, Sov. Phys. - JETP 20 762 (1965). 
[11] G. Sarma, J. Phys. Chem. Solids 24, 1092 (1963). 
[12] W. V. Liu and F. Wilczek, Phys. Rev. Lett. 90, 047002 (2003). 
M. M. Forbes et al., Phys.Rev.Lett. 94, 017001 (2005). S-T. Wu 
and S. Yip, Phys. Rev. A 67, 053603 (2003) 
[13] L. Radzihovsky, D. Sheehy, Rep. Prog. Phys. 73, 076501 
(2010). 

[14] S. Raghu, and S. Kivelson, Phys. Rev. B 83, 094518(201 1). 
[15] A. Bulgac, M. Forbes, and A. Schwenk, Phys. Rev. Lett. 97, 
020402 (2006); 

[16] Kelly R. Patton and D. E. Sheehy, Phys. Rev. A 83 051607 



5 



(2011). 

[17] J. K. Chin et al, Nature (London) 443, 961 (2006). 

[18] T. Stoferle et al, Phys. Rev. Lett. 96, 030401 (2006) 

[19] J. Hubbard, Proc. R. Soc. A 276, 238 (1963). 

[20] M. Inskin and C.A.R. Sa de Melo, Phys. Rev. Lett. 99, 080403 

(2007), and Phys. Rev. A 78, 013607 (2008). 
[21] H. J. Schulz, Europhys. Lett. 4, 609 (1987). 
[22] R. Shankar, Rev. Mod. Phys. 66, 129 (1994). 



[23] C. J. Halboth and W. Metzner, Phys. Rev. B 61, 7364 (2000). 
[24] D. Zanchi and H. J. Schulz, Phys. Rev. B 54, 9509; Europhys. 

Lett. 44, 235(1997); Phys. Rev. B 61, 13609 (2000). 
[25] S.-W. Tsai and J. B. Marston, Can. J. Phys. 79, 1463 (2001); C. 

Honerkamp, Phys. Rev. B 68, 104510 (2003). 
[26] W. Kohn and J.M. Luttinger, Phys.Rev.Lett. 15, 524 (1965). 



